Papers with audio-visual speech recognition
Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypotheses (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances have introduced GER frameworks that utilize LLMs to refine ASR outputs. |
| Approach: | They propose a framework that allows a large language model to compose independent N-best hypotheses from separate automatic speech recognition (ASR) and visual speech recognition models. |
| Outcome: | The proposed framework achieves 57.7% error rate gain over standard ASR baseline, compared to single-stream approaches that achieve only 10% gain. |
RUSAVIC Corpus: Russian Audio-Visual Speech in Cars (2022.lrec-1)
Copied to clipboard
| Challenge: | a new audio-visual speech corpus is recorded in a car environment and is noise-robust . currently there is no noiserobustic speech recognition system to be used in real-driving conditions. |
| Approach: | They propose to use a natural speech corpus recorded in a car environment to improve audio-based speech recognition in the presence of severe acoustic noise. |
| Outcome: | The proposed corpus is natural, controlled and adequate size to train state-of-the-art NN approaches. |
Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for audio-visual speech recognition use extra data to increase performance . a recent study shows that the use of unimodal self-supervised learning improves performance on multimodal tasks. |
| Approach: | They propose to use unimodal self-supervised learning to train AVSR models on unlabelled unilateral data. |
| Outcome: | The proposed model improves on lip reading sentences 2 by 30% even without an external language model. |
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment (2023.acl-long)
Copied to clipboard
| Challenge: | Speech Recognition often gets stuck in the lack of new domain utterances when training a model of new-domain speech. |
| Approach: | They propose a training system Open-modality Speech Recognition that enables zero-shot modality transfer . they use multi-modal alignment in phoneme space to maintain multi-modality alignment . |
| Outcome: | The proposed system achieves zero-shot modality transfer compared to existing methods . it achieves state-of-the-art performance on audio-visual speech recognition and lip-reading with 2.7% and 25.0%, respectively. |